{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "058cc67c-76d3-4898-bd49-8196a9976535",
   "metadata": {},
   "source": [
    "> - 越基础的，越需要重视，熟悉其原理，越基础的越会更广泛地出现。\n",
    "> - forward 前向计算过程，backward 反向梯度计算；\n",
    "> - 基于 torch（autograd 框架）的求导练习；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "76f78b47-162b-431c-ab07-92411ea7cbbb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime.datetime(2024, 4, 19, 20, 28, 0, 71905)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from datetime import datetime\n",
    "datetime.now()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "261233b6-180f-4bb1-a15b-c30bc89bea11",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x7c73300b69d0>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "\n",
    "torch.manual_seed(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab6e5e2d-72d2-4716-a0b7-ed194ea768a1",
   "metadata": {},
   "source": [
    "## forward"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c4a1b66-e0ed-4388-bc35-50bfaa623b5a",
   "metadata": {},
   "source": [
    "- randomly zeroes some of the elements of the **input tensor** with probability $p$.\n",
    "- the outputs are scaled by a factor of $\\frac1{1-p}$ during training. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5f02fe7c-5e20-407b-a19f-8f294ae360fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "m = nn.Dropout(0.4)\n",
    "input_ = torch.randn(5, requires_grad=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "d6938b35-eea4-4df5-821c-950b94e8a0fc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 1.5410, -0.2934, -2.1788,  0.5684, -1.0845], requires_grad=True)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "0dbb3ecb-630a-4f25-b5b8-94436db56429",
   "metadata": {},
   "outputs": [],
   "source": [
    "out = m(input_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1ee87871-0458-4878-81ee-0284eecd1388",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 0.0000, -0.4890, -0.0000,  0.9474, -1.8075], grad_fn=<MulBackward0>)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "18ce8cae-4277-4bdb-964d-f9afa34c7fb5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.489"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "-0.2934* (1/(1-0.4))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96c252f6-a1ef-4bf9-9af4-4aad4f637f17",
   "metadata": {},
   "source": [
    "## backward"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "180ab22f-617d-4a48-90c8-d6d36b2a0de2",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{split}\n",
    "y=x * mask \\frac{1}{1-p}\\\\\n",
    "y'=mask\\frac{1}{1-p}\n",
    "\\end{split}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ace7af32-223b-42a7-a630-aa3f4102806b",
   "metadata": {},
   "outputs": [],
   "source": [
    "y = out.sum()\n",
    "y.backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "74d6d0bf-94af-45bc-9118-66030d83b0fb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.0000, 1.6667, 0.0000, 1.6667, 1.6667])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2072e792-a6b1-4553-9dce-8361ece35692",
   "metadata": {},
   "source": [
    "## training/eval mode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "086da38a-9bb6-42e5-a6b2-8c13224b7ae6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "ad6449f3-fce2-4d6c-be8d-eb5f8eb2ed7e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dropout(p=0.4, inplace=False)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "d1817900-c7b2-4753-85c3-2dd23df11549",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dropout(p=0.4, inplace=False)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "aa9a27b7-cb67-43dd-91e4-cf88e6ae8bb3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "f8ab03f4-e335-4c8e-a6d2-aac978ad4cca",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([True, True, True, True, True])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m(input_) == input_"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
